{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "2eec5cc39a59"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4af014f3510c"
      },
      "source": [
        "<h1 align=\"center\"> <a href=\"../README.md\">Vertex AI: Gemini Evaluations Playbook </a><br>\n",
        "Prepare your environment</h1>\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69130f1b4423"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://art-analytics.appspot.com/r.html?uaid=G-FHXEFWTT4E&utm_source=aRT-gemini_evals_playbook_setup-from_notebook-colab&utm_medium=aRT-clicks&utm_campaign=gemini_evals_playbook_setup-from_notebook-colab&destination=gemini_evals_playbook_setup-from_notebook-colab&url=https%3A%2F%2Fcolab.sandbox.google.com%2Fgithub%2FGoogleCloudPlatform%2Fapplied-ai-engineering-samples%2Fblob%2Fmain%2Fgenai-on-vertex-ai%2Fgemini%2Fevals_playbook%2Fnotebooks%2F0_gemini_evals_playbook_setup.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://art-analytics.appspot.com/r.html?uaid=G-FHXEFWTT4E&utm_source=aRT-gemini_evals_playbook_setup-from_notebook-colab_ent&utm_medium=aRT-clicks&utm_campaign=gemini_evals_playbook_setup-from_notebook-colab_ent&destination=gemini_evals_playbook_setup-from_notebook-colab_ent&url=https%3A%2F%2Fconsole.cloud.google.com%2Fvertex-ai%2Fcolab%2Fimport%2Fhttps%3A%252F%252Fraw.githubusercontent.com%252FGoogleCloudPlatform%252Fapplied-ai-engineering-samples%252Fmain%252Fgenai-on-vertex-ai%252Fgemini%252Fevals_playbook%252Fnotebooks%252F0_gemini_evals_playbook_setup.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://art-analytics.appspot.com/r.html?uaid=G-FHXEFWTT4E&utm_source=aRT-gemini_evals_playbook_setup-from_notebook-github&utm_medium=aRT-clicks&utm_campaign=gemini_evals_playbook_setup-from_notebook-github&destination=gemini_evals_playbook_setup-from_notebook-github&url=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fapplied-ai-engineering-samples%2Fblob%2Fmain%2Fgenai-on-vertex-ai%2Fgemini%2Fevals_playbook%2Fnotebooks%2F0_gemini_evals_playbook_setup.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://art-analytics.appspot.com/r.html?uaid=G-FHXEFWTT4E&utm_source=aRT-gemini_evals_playbook_setup-from_notebook-vai_workbench&utm_medium=aRT-clicks&utm_campaign=gemini_evals_playbook_setup-from_notebook-vai_workbench&destination=gemini_evals_playbook_setup-from_notebook-vai_workbench&url=https%3A%2F%2Fconsole.cloud.google.com%2Fvertex-ai%2Fworkbench%2Fdeploy-notebook%3Fdownload_url%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fapplied-ai-engineering-samples%2Fblob%2Fmain%2Fgenai-on-vertex-ai%2Fgemini%2Fevals_playbook%2Fnotebooks%2F0_gemini_evals_playbook_setup.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d630feb56f1c"
      },
      "source": [
        "# Evals Playbook: Prepare your environment\n",
        "\n",
        "This notebook show you how to prepare the environment to run notebooks under [Gemini Evals Playbook](../README.md). The notebook performs following steps:\n",
        "\n",
        "- Enable required APIs in Google Cloud project\n",
        "- Required permissions and roles\n",
        "- Install required libraries using Poetry \n",
        "- Configure the necessary resources on Google Cloud such as BigQuery"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ecdb031df1d"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "Before you run this notebook, review the [README](../README.md) file of this repository to understand the features, architecture and data schema of Evals Playbook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "254fe5ce3b90"
      },
      "source": [
        "## 🎬 Getting Started\n",
        "\n",
        "The following steps are necessary to run this notebook, no matter what notebook environment you're using.\n",
        "\n",
        "If you're entirely new to Google Cloud, [get started here](https://cloud.google.com/docs/get-started).\n",
        "\n",
        "### Google Cloud Project Setup\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs\n",
        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "1. [Enable the Service Usage API](https://console.cloud.google.com/apis/library/serviceusage.googleapis.com)\n",
        "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)\n",
        "1. [Enable the Cloud Storage API](https://console.cloud.google.com/flows/enableapi?apiid=storage.googleapis.com)\n",
        "1. [Enable the Cloud BigQuery API](https://console.cloud.google.com/flows/enableapi?apiid=bigquery.googleapis.com)\n",
        "1. [Enable the Cloud Resource Manager API](https://console.cloud.google.com/flows/enableapi?apiid=cloudresourcemanager.googleapis.com)\n",
        "\n",
        "### Google Cloud Permissions\n",
        "\n",
        "**To run the complete Notebook, you will need to have the [Owner role](https://cloud.google.com/iam/docs/understanding-roles) for your project. At minimum, you need the following [roles](https://cloud.google.com/iam/docs/granting-changing-revoking-access)**:\n",
        "* **`roles/serviceusage.serviceUsageAdmin`** to enable APIs\n",
        "* **`roles/iam.serviceAccountAdmin`** to modify service agent permissions\n",
        "* **`roles/aiplatform.user`** to use AI Platform components\n",
        "* **`roles/storage.objectAdmin`** to modify and delete GCS buckets\n",
        "* **`roles/bigquery.user`** and **`roles/bigquery.dataViewer`** to query BigQuery tables\n",
        "* **`roles/bigquery.jobUser`** to run BigQuery jobs\n",
        "* **`roles/secretmanager.secretAccessor`** to access secret versions in Cloud Secret Manager"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46d25fe73955"
      },
      "source": [
        "### Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "18c6786bada3"
      },
      "outputs": [],
      "source": [
        "# Install poetry\n",
        "! pip uninstall poetry -y\n",
        "! pip install poetry --quiet\n",
        "\n",
        "# Run the poetry commands below to set up the environment\n",
        "! poetry lock # resolve dependencies (also auto create poetry venv if not exists)\n",
        "! poetry install --quiet # installs dependencies\n",
        "! poetry env info # displays the evn just created and the path to it"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfdc9c3907dc"
      },
      "source": [
        "### Restart Runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
        "\n",
        "You may see the restart reported as a crash, but it is working as-intended -- you are merely restarting the runtime.\n",
        "\n",
        "The restart might take a minute or longer. After it's restarted, continue to the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cfbcf60c4bd3"
      },
      "outputs": [],
      "source": [
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32e58b4d347b"
      },
      "source": [
        "### Authenticate\n",
        "\n",
        "If you're using Colab, run the code in the next cell. Follow the popups and authenticate with an account that has access to your Google Cloud [project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n",
        "\n",
        "If you're running this notebook somewhere besides Colab, make sure your environment has the right Google Cloud access. If that's a new concept to you, consider looking into [Application Default Credentials for your local environment](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev) and [initializing the Google Cloud CLI](https://cloud.google.com/docs/authentication/gcloud). In many cases, running `gcloud auth application-default login` in a shell on the machine running the notebook kernel is sufficient.\n",
        "\n",
        "More authentication options are discussed [here](https://cloud.google.com/docs/authentication)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2e53bd0701f4"
      },
      "outputs": [],
      "source": [
        "# Colab authentication.\n",
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()\n",
        "    print(\"Authenticated\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b5e4900ca75a"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
        "\n",
        "Make sure to change `PROJECT_ID` in the next cell. You can leave the values for `LOCATION`unless you have a specific reason to change them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "2e4c79ab6145"
      },
      "outputs": [],
      "source": [
        "# Define variables\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0cb20579b61c"
      },
      "source": [
        "### Create Google Cloud storage bucket\n",
        "Create or set Cloud Storage bucket name for Vertex AI staging and any other files relating to evals."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "b2080161acde"
      },
      "outputs": [],
      "source": [
        "STAGING_BUCKET = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
        "STAGING_BUCKET_URI = f\"gs://{STAGING_BUCKET}\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "04a525c5a3f5"
      },
      "outputs": [],
      "source": [
        "from google.cloud import storage\n",
        "\n",
        "storage_client = storage.Client(project=PROJECT_ID)\n",
        "\n",
        "# Check if bucket exists, create if not\n",
        "if not storage_client.bucket(STAGING_BUCKET).exists():\n",
        "    storage_client.create_bucket(STAGING_BUCKET)\n",
        "    print(f\"Bucket {STAGING_BUCKET} created!\")\n",
        "else:\n",
        "    print(f\"Bucket {STAGING_BUCKET} already exists\")\n",
        "\n",
        "bucket = storage_client.get_bucket(STAGING_BUCKET)\n",
        "# Verify the storage bucket project\n",
        "print(f\"Bucket is in the project {bucket.client.project}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d2e598c37687"
      },
      "source": [
        "### Enable required Google Cloud APIs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "72e0e0c5aaf9"
      },
      "outputs": [],
      "source": [
        "# Enable required APIs\n",
        "! gcloud services enable \\\n",
        "    iam.googleapis.com \\\n",
        "    storage-component.googleapis.com \\\n",
        "    compute.googleapis.com \\\n",
        "    aiplatform.googleapis.com \\\n",
        "    bigquery.googleapis.com \\\n",
        "    cloudresourcemanager.googleapis.com \\\n",
        "    --project $PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "00c3f047e40a"
      },
      "source": [
        "### Initialize Vertex AI SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4250f58d3442"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "module_path = os.path.abspath(os.path.join(\"..\"))\n",
        "sys.path.append(module_path)\n",
        "\n",
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION, staging_bucket=STAGING_BUCKET_URI)\n",
        "\n",
        "print(\"Vertex AI SDK initialized.\")\n",
        "print(f\"Vertex AI SDK version = {vertexai.__version__}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b69409b5203b"
      },
      "source": [
        "## ⛁ Create Data Schema"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9134be7f1331"
      },
      "source": [
        "Configure dataset name and table names to define data schema in BigQuery.\n",
        "\n",
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ You can leave the variables defined below as is, unless needed to change. ⚠️</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "bac3b4803ada"
      },
      "outputs": [],
      "source": [
        "# BigQuery datasets\n",
        "BQ_DATASET_ID = \"gemini_evals_playbook\"  # @param {type:\"string\"}\n",
        "BQ_LOCATION = \"US\"\n",
        "\n",
        "# DO NOT CHANGE\n",
        "BQ_TABLES_SQL_PATH = os.path.join(module_path, \"bigquery_sqls\", \"evals_bigquery.sql\")\n",
        "BQ_PREFIX = \"eval\"\n",
        "BQ_T_EVAL_TASKS = f\"{BQ_PREFIX}_tasks\"\n",
        "BQ_T_EXPERIMENTS = f\"{BQ_PREFIX}_experiments\"\n",
        "BQ_T_PROMPTS = f\"{BQ_PREFIX}_prompts\"\n",
        "BQ_T_DATASETS = f\"{BQ_PREFIX}_datasets\"\n",
        "BQ_T_EVAL_RUN_DETAILS = f\"{BQ_PREFIX}_run_details\"\n",
        "BQ_T_EVAL_RUNS = f\"{BQ_PREFIX}_runs\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "0f62baae35ba"
      },
      "outputs": [],
      "source": [
        "def setup_bigquery(bq_project_id, dataset_name, dataset_region, dry_run=False):\n",
        "    from google.cloud import bigquery\n",
        "\n",
        "    dataset_ref = f\"{bq_project_id}.{dataset_name}\"\n",
        "    job_config = bigquery.QueryJobConfig(dry_run=dry_run, default_dataset=dataset_ref)\n",
        "    client = bigquery.Client(\n",
        "        project=bq_project_id,\n",
        "        location=dataset_region,\n",
        "        default_query_job_config=job_config,\n",
        "    )\n",
        "    # create schema/dataset\n",
        "    try:\n",
        "        ddl = f\"\"\"\n",
        "            CREATE SCHEMA IF NOT EXISTS {dataset_name}\n",
        "            OPTIONS(\n",
        "                description=\"dataset for configuring Gemini evaluation tasks and storing evaluation results\",\n",
        "                {f\"location='{dataset_region}',\"if dataset_region else \"\"}\n",
        "                labels=[(\"tool\", \"vertexai-gemini-evals\")]\n",
        "            )\n",
        "        \"\"\"\n",
        "        print(\n",
        "            f\"Creating dataset {dataset_name} in project {bq_project_id}, if does not exists\"\n",
        "        )\n",
        "        print(ddl)\n",
        "        job = client.query(ddl)\n",
        "        results = job.result()\n",
        "        for result in results:\n",
        "            print(result)\n",
        "    except Exception as e:\n",
        "        print(\n",
        "            f\"Failed to create dataset {dataset_name} in project {bq_project_id} \\n{e}\"\n",
        "        )\n",
        "        raise e\n",
        "    # create tables\n",
        "    try:\n",
        "        print(f\"Creating tables in project {bq_project_id}, if does not exists.\")\n",
        "\n",
        "        ddl = open(BQ_TABLES_SQL_PATH).read()\n",
        "        print(ddl)\n",
        "\n",
        "        job = client.query(ddl)\n",
        "        result = job.result()\n",
        "        for result in results:\n",
        "            print(result)\n",
        "    except Exception as e:\n",
        "        print(f\"Failed to create tables in project {bq_project_id}\")\n",
        "        raise e"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3909e2b5ec74"
      },
      "outputs": [],
      "source": [
        "setup_bigquery(\n",
        "    bq_project_id=PROJECT_ID,\n",
        "    dataset_name=BQ_DATASET_ID,\n",
        "    dataset_region=BQ_LOCATION,\n",
        "    dry_run=False,\n",
        ")\n",
        "\n",
        "print(\n",
        "    \"Done! Created Bigquery dataset and tables to configure experiments and store eval results\"\n",
        ")\n",
        "print(\"You are ready to run the evaluations!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26f32bcabd11"
      },
      "source": [
        "## 💾 Save Configuration to File\n",
        "Save the configurations set in this notebook to  `config.ini`. The parameters from this file are used in subsequent notebooks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "35024d1e4bf2"
      },
      "outputs": [],
      "source": [
        "from utils.config import save_config\n",
        "\n",
        "save_config(\n",
        "    PROJECT_ID,\n",
        "    LOCATION,\n",
        "    STAGING_BUCKET,\n",
        "    STAGING_BUCKET_URI,\n",
        "    BQ_DATASET_ID,\n",
        "    BQ_LOCATION,\n",
        "    BQ_TABLES_SQL_PATH,\n",
        "    BQ_PREFIX,\n",
        "    BQ_T_EVAL_TASKS,\n",
        "    BQ_T_EXPERIMENTS,\n",
        "    BQ_T_PROMPTS,\n",
        "    BQ_T_DATASETS,\n",
        "    BQ_T_EVAL_RUN_DETAILS,\n",
        "    BQ_T_EVAL_RUNS,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "473a92000c02"
      },
      "source": [
        "### 🥁 If all the above steps are executed sucessfully, the following should be set up\n",
        "\n",
        "* GCP project and APIs to run the eval pipeline\n",
        "\n",
        "* All the required IAM permissions\n",
        "\n",
        "* Environment to run the notebooks\n",
        "\n",
        "* Bigquery datasets and tables to track evaluation results\n",
        "\n",
        "You can now proceed to run rest of the notebooks in Evals Playbook. Start with [1_gemini_evals_playbook_evaluate](1_gemini_evals_playbook_evaluate.ipynb) to design experiments, assess model performance on your generative AI tasks, and analyze evaluation results including side-by-side comparison of results across different experiments and runs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a618d4655148"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "0_gemini_evals_playbook_setup.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "gemini-evals",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
